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Abstract 

Gamma distributions, which contain the exponential as a special case, 
have a distinguished place in the representation of near-Poisson random- 
ness for statistical processes; typically, they represent distributions of 
spacings between events or voids among objects. Here we look at the 
properties of the Shannon entropy function and calculate its corresponding 
flow curves, relating them to examples of constrained degeneration from 
ordered processes. We consider also univariate and bivariate gamma, as 
well as WeibuU distributions since these include exponential distributions. 

Keywords: Shannon entropy, integral curves, gamma distribution, 
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1 Introduction 

The smooth family of gamma probability density functions is given by 

f ■.[0,^)^[0,^):x^ ^^'^ ^I,^>0. (1) 

T{k} 

Here /i is the mean, and the standard deviation a, given by k = (-)^, is pro- 
portional to the mean. Hence the coefficient of variation ^ is unity in the case 
that (jlj reduces to the exponential distribution. Thus, k = 1 corresponds to an 
underlying Poisson random process complementary to the exponential distribu- 
tion. When K < I the random variable X represents spacings between events 
that are more clustered than for a Poisson process and when k > 1 the spacings 
X are more uniformly distributed than for Poisson. The case when fi — n is a, 
positive integer and k = 2 gives the Chi-Squared distribution with n — 1 degrees 
of freedom; this is the distribution of '■""a-*" for variances of samples of size 

n taken from a Gaussian population with variance ctq. 

The gamma distribution has a conveniently tractable information geome- 
try [HE], and the Riemannian metric in the 2-dimensional manifold of gamma 
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Figure 1: Some examples of maximally extended geodesies passing through 
{ij.,k) = (1,1) in the gamma 2-manifold. 



distributions ([T]) is 





hj] (m, k) = = 



s&iog(r) 



(2) 



So the coordinates (/x, k) yield an orthogonal basis of tangent vectors, which is 
useful in calculations because then the arc length function is simply 




The system of geodesic equations is difficult to solve analytically but numerical 
solutions using the Mathematica programs of Gray [8] were obtained in [2]. 
Figure [T| shows a spray of some maximally extended geodesies emanating from 
the point {fi, k) — (1, 1). Geodesic curves have tangent vectors that are parallel 
along them and yield minimal information arc length distances in the gamma 
2-manifold. We note the following important uniqueness property: 

Theorem 1.1 (Hwang and Hu |9]) For independent positive random vari- 
ables with a common probability density function /, having independence of the 
sample mean and the sample coefficient of variation is equivalent to f being the 
gamma distribution. 

This property is one of the main reasons for the large number of applications 
of gamma distributions: many near-random natural processes have standard 
deviation approximately proportional to the mean |2]. Given a set of identically 
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distributed, independent data values Xi,X2, . . . , X„, the 'maximum likelihood' 
or 'maximum entropy' parameter values fi, k for fitting the gamma distribu- 
tion ([T]) are computed in terms of the mean and mean logarithm of the Xi by 
maximizing the likelihood function 

n 

Lf{fi,K) = Y[f{X,;n,K). 

i=l 

By taking the logarithm and setting the gradient to zero we obtain 

1 " 

A = X=-Y,X, (3) 



n 

i=l 



= logX-logX. (4) 



2 Gamma entropy flows 

Papoulis [14, Chapter 15 gives an account of the role of the Shannon entropy 
function in probability theory, stochastic processes and coding theory. The 
entropy of ([T|) is shown in Figure |2] using 

/>oo 

Sj = - / / log / : ^ K 

^ K-log(^^^+log{T{K))-{K-l)i^{^) (5) 

with gradient 

\H K J 

where -0 = is the digamma function. At fixed k, the entropy increases like 
log fi. At fixed mean fi, the maximum entropy is given by k = 1, the exponential 
distribution case of maximal disorder or chaos. 

Figure |2] on the right shows entropy as a contour plot with superimposed 
also some examples of integral curves of the entropy gradient flow field, namely 
curves c satisfying 

c:[0,oo)->M2:c(t)=V5/|(^(,)). (7) 

By inspection, we can see that the entropy gradient components are each in one 
variable only and in particular the first component has solution 





Figure 2: Shannon entropy Junction Sf for the gamma family as a surface with 
respect to mean fi and n (left) and as a contour plot with entropy gradient flow 
and integral curves (right). The asymptote is k = I, the exponential case of 
maximum disorder. 

so the mean increases exponentially with time. Such curves represent typical 
trajectories for processes subordinate to gamma distributions; the processes be- 
come increasingly disordered as k ^ 1. The entropy gradient curves correspond 
to systems with external input — the mean increases as disorder increases. The 
asymptote is k = 1, the exponential case of maximum disorder. Conversely, 
the reverse direction of the curves corresponds to evolution from total disorder 
to other states (clustered for k < 1, and smoothed out, 'more crystal-like', for 
K > 1) while the mean is allowed to reduce — somewhat like the situation after 
the Big Bang, see Dodson [3]. 



3 Constrained degeneration of order 

Lucarini |12j effectively illustrated the degeneration of order in his perturbations 
of the simple 3D cubic crystal lattices (SC, BCC, FCC) by an increasing spatial 
Gaussian noise. Physically, the perturbing spatial noise intensity corresponds 
somewhat to a lattice temperature in the structural symmetry degeneration. 
With rather moderate levels of noise, quite quickly the three tessellations be- 
came indistinguishable. In the presence of intense noise they all converged to the 
3D Poisson-Voronoi tessellations, for which exact analytic results are known ^ . 
Moreover, in all cases the gamma distribution was an excellent model for the 
observed probability density functions of all metric and topological properties. 
See also Ferenc and Neda ,6^ for some analytic approximations using gamma 
distributions for two and three dimensional Poisson-Voronoi cell size statistics. 
Lucarini provided plots showing the evolution of the mean and standard devi- 
ation of these properties as they converge asymptotically towards the Poisson- 
Voronoi case, illustrating the degeneration of crystallinity from k ^ oo to lower 
values. 
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100,000-200,000 


14.49 


11.93 


0.823 


1.67 


200,000-300,000 


15.05 


12.48 


0.830 


1.67 


300,000-400,000 


15.43 


12.78 


0.829 


1.64 


400,000-500,000 


15.64 


12.97 


0.829 


1.64 


500,000-600,000 


15.88 


13.23 


0.833 


1.62 


600,000-700,000 


16.08 


13.36 


0.831 


1.62 


700,000-800,000 


16.20 


13.51 


0.834 


1.62 


800,000-900,000 


16.35 


13.59 


0.831 


1.61 


900,000-1,000,000 


16.46 


13.75 


0.835 


1.60 


1-10,000,000 


17.81 


15.01 


0.843 


1.56 


1-100,000,000 


20.07 


16.97 


0.846 


1.34 



Table 1: Statistical properties of the spacings between consecutive prime num- 
bers: mean /ip, standard deviation ap, coefficient of variation cvp — maxi- 
mum likelihood gamma parameter Kp, for each of the first ten blocks of 100,000 
primes, and the overall data for the first 10 million primes and the first 100 
million primes. 

Of course, the constraint of remaining tessellations, albeit highly disordered 
ones, precludes convergence down to the maximum entropy hmit k = 1. In fact 
the limiting values are k « 16 for number of vertices and the same for num- 
ber of edges and k « 22 for the number of faces; actually these are discrete 
random variables and the gamma is not appropriate. However, for the posi- 
tive real random variables, polyhedron volume in the limit has k w 5.6 and 
polygon face area k « 16. Lucarini [llj had reported similar findings for the 
2D case of perturbations of the three regular tessellations of the plane: square, 
hexagonal and triangular. There also the gamma distribution gave a good fit 
for the distributions during the degeneration of the regular tessellations to the 
2D Poisson-Voronoi case; the limiting values were k « 16 for the perimeter of 
polygons and k « 3.7 for areas. 

We can give another perspective on the level of constraint persistent in the 
limits for these disordered tessellations: for infinite random matrices the best 
fitting gamma distributions for eigenvalue spacings have k — 2.420, 4.247, 9.606 
respectively for orthogonal, unitary and symplectic Gaussian ensembles [2]. 
These values in a sense indicate the increasing statistical constraints of alge- 
braic relations as the ensembles change through orthogonality, unitarity and 
symplectivity. 

In the context of analytic number theory, gamma distributions give approx- 
imate distributions for the the reported data on spacings between zeros of the 
Riemann zeta function 13J. The best fit gamma distributions to spacings be- 
tween the first two million zeros of the Riemann zeta function has k w 5.6 [2\. 

The spacings between consecutive prime numbers in successive blocks have 
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Figure 3: Frequency histogram of the spacings between the first 100 million 
primes and the maximum likelihood gamma frequencies (points) with np = 1.34. 
The early peaks in rank order are at spacings of 6, 12, 10, 18, 2 and ^. The 
mean spacing fip — 20.07 occurs close to rank order 11. 



surprisingly stable means and standard deviations, as we see in Table [T] This 
gives the mean /ip, standard deviation up, coefficient of variation cvp, maxi- 
mum likelihood gamma parameter Kp for each of the first ten blocks of 100,000 
primes, and the same data for the first 100 million primes. Of course, the gamma 
distribution is not a good fit, as we can see in Figure |3] for the first 100 million 
primes. From Table (T) the mean drifts up with the local mean size, as we would 
expect from the Prime Number Theorem, but so also does the standard devi- 
ation in proportion, hence keeping the coefficient of variation nearly constant. 
This stability suggests that there should be some qualitative number theoretic 
property which is being reflected and that the distribution of spacings among 
these early primes is somewhat close to a Poisson process (k = 1). 

For the SARS disease epidemic outbreak [3J US] the gamma distribution gave 
a good model and the infection process was approximated by k w 3, cf. ^. 



4 Bivariate gamma processes 

Next we consider the bivariate case of a pair of coupled gamma processes. The 
McKay family can be thought of as giving the probability density for the two 
random variables X and Y — X + Z where X and Z both have gamma dis- 
tributions. This smooth bivariate family M of density functions is defined in 
the positive octant of random variables < x < y < oo with parameters 
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ai,c, a2 G M"*" and probability density functions 

m{x,y) = — — — - — ^ 

r(ai)r(a2) 

The marginal density functions, of X and Y are: 



(8) 



nixix) 
■myiy) 



r(«i) 



a: > 



g(Qi+Q2)y(ai+a2)-lg-ca 



T{ai + 012) 



y>0. 



(9) 
(10) 



Note that we cannot have both marginal distributions exponential. The covari- 
ance and correlation coefficient of X and Y are: 



0'12 



ai 



and p{X, Y) = 



ai 



Unlike other bivariate gamma families, the McKay information geometry is 
surprisingly tractable and there are a number of applications discussed in Arwini 
and Dodson In fact the parameters ai,c,a2 G are natural coordinates 
for the 3-dimensional manifold M of this family. The Riemannian information 
metric is 



[gij{ai,c,a2)] 



_ 1 

C 









V''(a2) 



(11) 



It is difficult to present graphics of curves in the McKay 3-manifold M, but it 
has an interesting 2-dimensional submanifold Mi <Z M: ai = \. The density 
functions are of form: 



h{x,y\ l,c, 02) 



(12) 



defined onO<a;<y<oo with parameters c, 02 G . The correlation 
coefficient and marginal functions of X and Y are given by: 



1 



\/l + "2 



P{X,Y) = 
hx{x) = 
hy{y) = 



(13) 
(14) 
(15) 



In fact a2 — J2 , which in applications would give a measure of the variability 
not due to the correlation. The matrix of metric components [gij] on Mi is 

1+02 _ 1 

4 V^'("2) 



(16) 




Some geodesies emanating from {a2,c) = (1, 1) G Mi are shown on the left of 
Figure |4] The density functions can be presented also in terms of the positive 
parameters (ai, cri2, 0:2) where ai2 is the covariance of X and Y 



m,{x,y;ai,ai2,a2) 
■mx{x) 



r(ai)r(a2) 

"12 

, a; > 



r(«i) 

(^) '°^2°'"' j;(«i+"2)-l( 

T{ai + a2) 



(17) 

(18) 

y>0- (19) 



The entropy function is 
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McKay entropy S, 




Figure 5: Surface representation of the Shannon entropy function Sm for the 
submanifold Mi of the McKay family with respect to parameter and correla- 
tion coefficient p (left), and contour plot (right). Superimposed also are gradient 
flow arrows and the thick curve is the locus of maximum entropy. The two thin- 
ner curves show the loci o/ai = 1 (upper curve) and ai +a2 = 1 (lower curve), 
which correspond, respectively, to Poisson random processes for the X and Y 
variables. 



(ai,c, 02) 
K 

On Ml 

Sm\Mi 



aic 



— Ql — 1 



K 



= log 



\Mi 



r(ai)r(a2) 



+ {ai - l)ij{ai) + {a2 - l)'(/'(a2) - ("i + (^2). 



+ {a2 ~ l)-0(a2) - (1 + 012) 

tp{a2)ia2 - 1) + a2 + 2 



(20) 



(21) 



(V'(a2)(a2 - 1) - 1) 



On the right of Figure [4] is a contour plot of the entropy showing its gradient 
field and some integral curves. It may be helpful to express the entropy in terms 
of a2 and p, which gives 



Sn 



I Ml 



4p4 



log 



V(r(a2)) 



+ (a2-l)^(a2)-(l + «2) .(22) 



The McKay entropy in Mi with respect to a2, p is shown in Figure [5] (left) with 
the gradient flow on a contour plot (right) together with the approximate locus 
curve of maximum entropy. The two thinner curves show the loci of ai = 1 
(upper curve) and ai +a2 = 1 (lower curve), which correspond, respectively, to 
Poisson random processes for the X and Y variables. 

Qualitatively, what we may see in Figure |5] is that for correlated random 



10 



On the entropy Rows to disorder 




variables X and Y subordinate to the bivariate gamma density (12 1, the maxi- 
mum entropy locus is roughly hyperbolic. The maximum entropy curve is rather 
insensitive to the correlation coefficient p when a2 > I and the difference Y — X 
is dispersed more evenly than Poisson. When Y — X is actually Poisson ran- 
dom, with a2 = 1, the critical value is at p « 0.355. However, as a2 reduces 
further — corresponding to clustering ofY — X values — so the locus turns rapidly 
to increasing correlation. If we take the situation of a bivariate gamma process 
with constant marginal mean values, then the McKay probability density has 
constant correlation coefficient p; in this case the gradient flow lies along lines 
of constant a2- 

Dodson [5l gives an application to an epidemic model in which the latency 
and infectivity for individuals in a population are jointly distributed properties 
controlled by a bivariate gamma distribution. 



5 Weibull processes 

Like the gamma family, the Weibull family of distributions contains the ex- 
ponential distribution as a special case; it has wide application in models for 
reliability and lifetime statistics for random variable t > 0. The probability 
density function can be presented in terms of positive parameters ^, /3 

w:R+ ^ R:t^^ (lY^'e'^'^^^' . (23) 



In applications of (23), reliability R(t) is the probability of survival to time t 
and it is related to the failure rate Z(t) at time t through 

R{t) = w{t) dt = and Z{t) ^ ill = /3 t^'K (24) 
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The WeibuU mean, standard deviation and entropy are 

1 



/x„ = ^r(l + ^) (25) 




(26) 

SU^,f3) = -log(/3)-log( iV?+7+l (27) 



In (27 1, 7 is the Euler constant, of value approximately 0.577. 



In case /3 = ^ for positive integer n, then the coefficient of variation is 



a^^ /(2n)! 



and we see that the case (3=1 reduces ( p3| ) to the exponential density with 
Mto = = ^ and hence unit coefficient of variation. Figure [6] shows a surface 
plot of the WeibuU entropy and a corresponding contour plot with gradient 
flow and some integral curves. There is a resemblance to the gamma distribution 
entropy in Figure [2] and from (28 1 again here we have /i^(i) = /iu,(0)e*, but in 



the WeibuU case the asymptotic curve has (3 = ^ ~ 0.577, which corresponds 
to a process with coefficient of variation « 4.65 compared with unity for the 
exponential distribution and Poisson randomness. 
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